{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c7170d8",
   "metadata": {},
   "source": [
    "## Storing data in ultra-fast HDF5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efd7eeae",
   "metadata": {},
   "outputs": [],
   "source": [
    "!conda install -c conda-forge pytables -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a2b67c34",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "533516f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from IPython.display import Markdown, display\n",
    "from openbb import obb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ddbd8c41",
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings(\"ignore\")\n",
    "obb.user.preferences.output_type = \"dataframe\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e63f6079",
   "metadata": {},
   "outputs": [],
   "source": [
    "STOCKS_DATA_STORE = \"stocks.h5\"\n",
    "FUTURES_DATA_STORE = \"futures.h5\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0565ca7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ticker = \"SPY\"\n",
    "root = \"ES\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a050185",
   "metadata": {},
   "source": [
    "Fetches historical price data for the equity \"SPY\" starting from 2021-01-01 using the \"yfinance\" provider and stores it in 'spy_equity'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3843471e",
   "metadata": {},
   "outputs": [],
   "source": [
    "spy_equity = obb.equity.price.historical(\n",
    "    ticker, start_date=\"2021-01-01\", provider=\"yfinance\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e4371ac",
   "metadata": {},
   "source": [
    "Fetches options chains for the equity \"SPY\" using the \"cboe\" provider and stores it in 'spy_chains'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4de126e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "spy_chains = obb.derivatives.options.chains(ticker, provider=\"cboe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c21cdc6c",
   "metadata": {},
   "source": [
    "Gets the unique expiration dates from 'spy_chains'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9ace1bce",
   "metadata": {},
   "outputs": [],
   "source": [
    "spy_expirations = spy_chains.expiration.astype(str).unique().tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9f31f36",
   "metadata": {},
   "source": [
    "Fetches historical price data for a specific option of \"SPY\" starting from 2021-01-01 using the \"yfinance\" provider and stores it in 'spy_historic'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bb2af771",
   "metadata": {},
   "outputs": [],
   "source": [
    "spy_historic = obb.equity.price.historical(\n",
    "    ticker + spy_expirations[-10].replace(\"-\", \"\")[2:] + \"C\" + \"00400000\",\n",
    "    start_date=\"2021-01-01\",\n",
    "    provider=\"yfinance\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "060014b4",
   "metadata": {},
   "source": [
    "Save stock data to HDF5 store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fb92fb23",
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.HDFStore(STOCKS_DATA_STORE) as store:\n",
    "    store.put(\"equities/spy/stock_prices\", spy_equity)\n",
    "    store.put(\"equities/spy/options_prices\", spy_historic)\n",
    "    store.put(\"equities/spy/chains\", spy_chains)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9553bd5f",
   "metadata": {},
   "source": [
    "Save futures data to HDF5 store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f38077bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.HDFStore(FUTURES_DATA_STORE) as store:\n",
    "    for i in range(24, 31):\n",
    "        expiry = f\"20{i}-12\"\n",
    "        df = obb.derivatives.futures.historical(\n",
    "            symbol=[root],\n",
    "            expiry=expiry,\n",
    "            start_date=\"2021-01-01\",\n",
    "        )\n",
    "        df.rename(columns={\"close\": expiry}, inplace=True)\n",
    "        prices = df[expiry]\n",
    "\n",
    "        store.put(f\"futures/{root}/{expiry}\", prices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41e73220",
   "metadata": {},
   "source": [
    "Load stock data from HDF5 store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a26e2598",
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.HDFStore(STOCKS_DATA_STORE) as store:\n",
    "    spy_prices = store[\"equities/spy/stock_prices\"]\n",
    "    spy_options = store[\"equities/spy/options_prices\"]\n",
    "    spy_chains = store[\"equities/spy/chains\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c39bb5b",
   "metadata": {},
   "source": [
    "Load futures data from HDF5 store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eac7c0f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.HDFStore(FUTURES_DATA_STORE) as store:\n",
    "    es_prices = store[f\"futures/{root}/2024-12\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8fa39c64",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>split_ratio</th>\n",
       "      <th>dividend</th>\n",
       "      <th>capital_gains</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-01-04</th>\n",
       "      <td>375.309998</td>\n",
       "      <td>375.450012</td>\n",
       "      <td>364.820007</td>\n",
       "      <td>368.790009</td>\n",
       "      <td>110210800</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-05</th>\n",
       "      <td>368.100006</td>\n",
       "      <td>372.500000</td>\n",
       "      <td>368.049988</td>\n",
       "      <td>371.329987</td>\n",
       "      <td>66426200</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-06</th>\n",
       "      <td>369.709991</td>\n",
       "      <td>376.980011</td>\n",
       "      <td>369.119995</td>\n",
       "      <td>373.549988</td>\n",
       "      <td>107997700</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-07</th>\n",
       "      <td>376.100006</td>\n",
       "      <td>379.899994</td>\n",
       "      <td>375.910004</td>\n",
       "      <td>379.100006</td>\n",
       "      <td>68766800</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-08</th>\n",
       "      <td>380.589996</td>\n",
       "      <td>381.489990</td>\n",
       "      <td>377.100006</td>\n",
       "      <td>381.260010</td>\n",
       "      <td>71677200</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-12</th>\n",
       "      <td>541.630005</td>\n",
       "      <td>544.119995</td>\n",
       "      <td>540.299988</td>\n",
       "      <td>541.359985</td>\n",
       "      <td>63251300</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-13</th>\n",
       "      <td>543.150024</td>\n",
       "      <td>543.330017</td>\n",
       "      <td>539.590027</td>\n",
       "      <td>542.450012</td>\n",
       "      <td>44760900</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-14</th>\n",
       "      <td>540.880005</td>\n",
       "      <td>542.809998</td>\n",
       "      <td>539.849976</td>\n",
       "      <td>542.780029</td>\n",
       "      <td>40089900</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-17</th>\n",
       "      <td>542.080017</td>\n",
       "      <td>548.530029</td>\n",
       "      <td>541.609985</td>\n",
       "      <td>547.099976</td>\n",
       "      <td>55839500</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-18</th>\n",
       "      <td>547.159973</td>\n",
       "      <td>548.619995</td>\n",
       "      <td>546.729980</td>\n",
       "      <td>548.489990</td>\n",
       "      <td>41342200</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>870 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  open        high         low       close     volume  \\\n",
       "date                                                                    \n",
       "2021-01-04  375.309998  375.450012  364.820007  368.790009  110210800   \n",
       "2021-01-05  368.100006  372.500000  368.049988  371.329987   66426200   \n",
       "2021-01-06  369.709991  376.980011  369.119995  373.549988  107997700   \n",
       "2021-01-07  376.100006  379.899994  375.910004  379.100006   68766800   \n",
       "2021-01-08  380.589996  381.489990  377.100006  381.260010   71677200   \n",
       "...                ...         ...         ...         ...        ...   \n",
       "2024-06-12  541.630005  544.119995  540.299988  541.359985   63251300   \n",
       "2024-06-13  543.150024  543.330017  539.590027  542.450012   44760900   \n",
       "2024-06-14  540.880005  542.809998  539.849976  542.780029   40089900   \n",
       "2024-06-17  542.080017  548.530029  541.609985  547.099976   55839500   \n",
       "2024-06-18  547.159973  548.619995  546.729980  548.489990   41342200   \n",
       "\n",
       "            split_ratio  dividend  capital_gains  \n",
       "date                                              \n",
       "2021-01-04          0.0       0.0            0.0  \n",
       "2021-01-05          0.0       0.0            0.0  \n",
       "2021-01-06          0.0       0.0            0.0  \n",
       "2021-01-07          0.0       0.0            0.0  \n",
       "2021-01-08          0.0       0.0            0.0  \n",
       "...                 ...       ...            ...  \n",
       "2024-06-12          0.0       0.0            0.0  \n",
       "2024-06-13          0.0       0.0            0.0  \n",
       "2024-06-14          0.0       0.0            0.0  \n",
       "2024-06-17          0.0       0.0            0.0  \n",
       "2024-06-18          0.0       0.0            0.0  \n",
       "\n",
       "[870 rows x 8 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>split_ratio</th>\n",
       "      <th>dividend</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2022-09-13</th>\n",
       "      <td>64.449997</td>\n",
       "      <td>64.449997</td>\n",
       "      <td>64.449997</td>\n",
       "      <td>64.449997</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-09-15</th>\n",
       "      <td>61.419998</td>\n",
       "      <td>61.419998</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-09-16</th>\n",
       "      <td>60.830002</td>\n",
       "      <td>60.830002</td>\n",
       "      <td>58.000000</td>\n",
       "      <td>58.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-09-19</th>\n",
       "      <td>57.669998</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>57.570000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>411</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-09-20</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>56.099998</td>\n",
       "      <td>124</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-07</th>\n",
       "      <td>147.220001</td>\n",
       "      <td>147.220001</td>\n",
       "      <td>147.220001</td>\n",
       "      <td>147.220001</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-12</th>\n",
       "      <td>154.710007</td>\n",
       "      <td>154.710007</td>\n",
       "      <td>154.009995</td>\n",
       "      <td>154.009995</td>\n",
       "      <td>131</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-13</th>\n",
       "      <td>154.000000</td>\n",
       "      <td>154.000000</td>\n",
       "      <td>151.520004</td>\n",
       "      <td>153.130005</td>\n",
       "      <td>16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-17</th>\n",
       "      <td>153.850006</td>\n",
       "      <td>153.850006</td>\n",
       "      <td>153.850006</td>\n",
       "      <td>153.850006</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-06-18</th>\n",
       "      <td>158.720001</td>\n",
       "      <td>158.720001</td>\n",
       "      <td>158.720001</td>\n",
       "      <td>158.720001</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>345 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  open        high         low       close  volume  \\\n",
       "date                                                                 \n",
       "2022-09-13   64.449997   64.449997   64.449997   64.449997      23   \n",
       "2022-09-15   61.419998   61.419998   61.000000   61.000000       5   \n",
       "2022-09-16   60.830002   60.830002   58.000000   58.000000       2   \n",
       "2022-09-19   57.669998   60.000000   57.570000   60.000000     411   \n",
       "2022-09-20   60.000000   60.000000   56.000000   56.099998     124   \n",
       "...                ...         ...         ...         ...     ...   \n",
       "2024-06-07  147.220001  147.220001  147.220001  147.220001       1   \n",
       "2024-06-12  154.710007  154.710007  154.009995  154.009995     131   \n",
       "2024-06-13  154.000000  154.000000  151.520004  153.130005      16   \n",
       "2024-06-17  153.850006  153.850006  153.850006  153.850006       2   \n",
       "2024-06-18  158.720001  158.720001  158.720001  158.720001       1   \n",
       "\n",
       "            split_ratio  dividend  \n",
       "date                               \n",
       "2022-09-13          0.0       0.0  \n",
       "2022-09-15          0.0       0.0  \n",
       "2022-09-16          0.0       0.0  \n",
       "2022-09-19          0.0       0.0  \n",
       "2022-09-20          0.0       0.0  \n",
       "...                 ...       ...  \n",
       "2024-06-07          0.0       0.0  \n",
       "2024-06-12          0.0       0.0  \n",
       "2024-06-13          0.0       0.0  \n",
       "2024-06-17          0.0       0.0  \n",
       "2024-06-18          0.0       0.0  \n",
       "\n",
       "[345 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "date\n",
       "2000-09-18    1467.5\n",
       "2000-09-19    1478.5\n",
       "2000-09-20    1469.5\n",
       "2000-09-21    1469.5\n",
       "2000-09-22    1468.5\n",
       "               ...  \n",
       "2024-06-13    5438.5\n",
       "2024-06-14    5437.5\n",
       "2024-06-17    5478.5\n",
       "2024-06-18    5491.0\n",
       "2024-06-19    5493.0\n",
       "Name: 2024-12, Length: 6000, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(spy_prices)\n",
    "display(spy_options)\n",
    "display(es_prices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1e98b11",
   "metadata": {},
   "source": [
    "**Jason Strimpel** is the founder of <a href='https://pyquantnews.com/'>PyQuant News</a> and co-founder of <a href='https://www.tradeblotter.io/'>Trade Blotter</a>. His career in algorithmic trading spans 20+ years. He previously traded for a Chicago-based hedge fund, was a risk manager at JPMorgan, and managed production risk technology for an energy derivatives trading firm in London. In Singapore, he served as APAC CIO for an agricultural trading firm and built the data science team for a global metals trading firm. Jason holds degrees in Finance and Economics and a Master's in Quantitative Finance from the Illinois Institute of Technology. His career spans America, Europe, and Asia. He shares his expertise through the <a href='https://pyquantnews.com/subscribe-to-the-pyquant-newsletter/'>PyQuant Newsletter</a>, social media, and has taught over 1,000+ algorithmic trading with Python in his popular course **<a href='https://gettingstartedwithpythonforquantfinance.com/'>Getting Started With Python for Quant Finance</a>**. All code is for educational purposes only. Nothing provided here is financial advise. Use at your own risk."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
